Skip to content

test: Change the file size in the override retry test so the poller sees the malformed write - #576

Merged
kinyoklion merged 1 commit into
feat/overridesfrom
rlamb/overrides-python-retry-test-size
Nov 12, 2026
Merged

kinyoklion merged 1 commit into
feat/overridesfrom
rlamb/overrides-python-retry-test-size

Conversation

@kinyoklion

@kinyoklion kinyoklion commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Requirements

  • I have added test coverage for new or changed functionality
  • I have followed the repository's pull request submission guidelines
  • I have validated my changes against all supported platform versions
  • Sync/async parity: matching changes made to async_* siblings (or N/A). N/A: a test-only change to the file override source tests.

Related issues

test_source_retries_a_failed_load_without_a_change_signal failed on windows (3.10) in the CI of #572, with AssertionError: the load failure was not logged. The test passed on every other Windows and Linux version.

Describe the solution you've provided

The test writes valid content, then malformed content, then fixed content. The fixed content keeps the malformed content's size and modification time, so that only the automatic retry can recover. The initial content had that same size too. The poller compares a file's modification time and size. Where file times are coarse, as on Windows, the initial and malformed writes can share a modification time, so the poller saw no change. Nothing was then logged, and the test timed out.

The initial content now has a different size from the malformed content, so the poller always sees the malformed write. The part of the test that depends on an unchanged size and time is unchanged.

To confirm the cause, I patched _observe to round modification times down to whole seconds and ran the test. It failed after its 10-second timeout without this change, and it passes with it. The whole test file passes normally (72 tests), and isort and pycodestyle are clean.

Describe alternatives you've considered

Setting the file's modification time after the malformed write. Changing the size states the intent in the content itself, and needs no extra system call.

Additional context

This merges cleanly with #574, which adds tests to the same file.

Opened from termag


Note

Overview
Stabilizes test_source_retries_a_failed_load_without_a_change_signal on Windows by changing the initial override JSON so its file size differs from the later malformed payload.

The test still relies on malformed and fixed content having the same length and mtime so recovery must come from automatic retry, not polling. On platforms with coarse mtimes, the old initial content could match the malformed write on size and time, so the poller missed the bad edit and the expected load-failure log never appeared. Comments were added to document that intent.

Reviewed by Cursor Bugbot for commit 0d9ba1c. Bugbot is set up for automated code reviews on this repo. Configure here.

…ees the malformed write

The initial and malformed contents had the same size. Where file times are coarse, as on Windows, both writes can share a modification time, so the poller saw no change, nothing was logged, and the test timed out. The initial content now has a different size.
@kinyoklion
kinyoklion merged commit 69a55e5 into feat/overrides Nov 12, 2026
18 checks passed
@kinyoklion
kinyoklion deleted the rlamb/overrides-python-retry-test-size branch November 12, 2026 17:17
kinyoklion added a commit that referenced this pull request Nov 12, 2026
…to a touch (#577)

<!-- TERMAG_PR_BODY_BEGIN -->
**Requirements**

- [x] I have added test coverage for new or changed functionality
- [x] I have followed the repository's [pull request submission
guidelines](../blob/main/CONTRIBUTING.md#submitting-pull-requests)
- [ ] I have validated my changes against all supported platform
versions
- [x] Sync/async parity: matching changes made to `async_*` siblings (or
N/A). N/A: the file watcher serves both clients.

**Related issues**

This resolves the SDK findings of the Windows stability assessment for
the overrides work:
[plans/python-overrides-windows-test-stability.md](https://github.com/launchdarkly/sdk-scratchpad/blob/main/plans/python-overrides-windows-test-stability.md)
in sdk-scratchpad.

- Finding 1, the flaky retry test, is fixed by #576.
- Findings 2, 3 and 4 are fixed here.

**Describe the solution you've provided**

All three changes are in `Watcher`, in
`ldclient/impl/integrations/files/filedata.py`, which the file-based
override source uses in watching mode.

1. **watchdog 7 delays Windows events by a second, and starves them in a
busy directory** (Finding 2).
- watchdog 7.0.0 (2026-10-31) changed its Windows reader. It now waits
for the observer timeout after the first notification before delivering
anything, and it delivers nothing while notifications keep arriving
within that timeout.
- With the default one-second timeout, a change to an override file was
reported about a second late on Windows. It was not reported at all
while another file in the same directory, such as a log, changed more
often than once a second.
- The assessment measured, on Windows Server 2025 with watchdog 7.1.0: a
median latency of 1003 ms, and three of three changes not reported
within 8 s next to a file appended every 200 ms. With a 100 ms timeout:
102 to 110 ms, and 3 of 3 reported.
- The watcher now builds the observer with `timeout=0.1`. On Linux, the
timeout only sets how often the idle inotify loop wakes.
- Python 3.11 and later install watchdog 7 under `watchdog>=3.0.0`, so
this affects any Windows deployment that selects
`ChangeDetection.WATCHING`.
2. **A touch no longer reloads with watchdog 7** (Finding 3).
- watchdog 7 reports a change to a file's metadata, such as a new
modification time from `os.utime`, as an `attrib` event. Earlier
versions reported it as `modified`.
- The watcher ignored `attrib`, so touching a file to force a reload
stopped working in watching mode, while it still works in polling mode.
- `attrib` now counts as a change. A plain read does not produce it, so
the reloads stay quiet. The reloader's content digest still skips
applying an unchanged file.
3. **watchdog 6 on Windows can miss a change right after a watch
starts** (Finding 4).
- watchdog 6.0.0, the last version that supports Python 3.10, returns
from starting a watch after a fixed 10 ms. Its reader thread may not yet
have asked for the first notification, and anything before that request
is lost.
- The assessment found that a deletion made immediately after the
watcher was created was reported 0 of 10 times, and 10 of 10 times after
one thread yield.
   - The watcher now yields once after each watch it sets up.

Tests in `test_filedata.py`:
- a change is reported within 0.5 s;
- a change is reported within 2 s while another file in the directory is
appended to five times a second;
- a change of the modification time alone is reported.

**Describe alternatives you've considered**

- Pinning `watchdog<7` in the `test-filesource` extra. Applications
bring their own watchdog, so a pin would only hide the behavior from CI.
- Running a slow safety poll under watching mode, so that a lost or
starved notification is bounded. That is a design decision for the spec,
not part of this fix.
- Documenting that watching mode ignores a touch, instead of reacting to
`attrib`. That would leave watching and polling behaving differently for
the same file.

**Additional context**

Testing:
- On Linux with watchdog 7.1.0, the touch test fails without this change
and passes with it.
- The latency and busy-directory tests pass on Linux either way, because
inotify delivers each event at once. Windows CI runs them against
watchdog 7, which they target.
- Every watcher test passes with watchdog 6.0.0 and with watchdog 7.1.0.
That includes the existing test that a read alone does not signal.
- The file data and override source modules pass on watchdog 7.1.0 (263
tests).
- The full suite passes (2,120 tests), without the Redis, DynamoDB and
Consul store tests, which need servers that were not running.
- `mypy`, `isort` and `pycodestyle` are clean.

This PR merges cleanly with #574 and #576.

Not in this PR, from Finding 6 of the assessment. None of these come
from the overrides work:
- a latency assertion in `test_async_migrator.py` that is too tight for
Windows clocks;
- a streaming retry test that is flaky on `main`;
- the big segments status-polling contract test, which turns Linux CI on
`feat/overrides` red in 4 of 5 runs and needs its own investigation
before the merge into `main`.

The assessment also recommends reporting the watchdog 7 behavior
upstream; that is not done yet.
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=python-server-sdk -->
<sub>Opened from termag</sub>

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Updates the file **`Watcher`** in watching mode so override/data
reloads behave reliably with **watchdog 6–7**, especially on Windows.
> 
> The observer now uses a **100 ms** delivery timeout instead of
watchdog’s 1 s default, so changes are reported quickly and are not
indefinitely delayed when another file in the same directory keeps
notifying. **`attrib`** is treated as a content-relevant event so a
**mtime-only touch** triggers a reload again under watchdog 7. After
each directory watch is scheduled, a **thread yield** (`time.sleep(0)`)
avoids missing an immediate post-start change on watchdog 6 for Windows.
> 
> New watcher tests cover sub-second latency, detection alongside a busy
log file, and mtime-only updates.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
546a7f5. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
kinyoklion added a commit that referenced this pull request Nov 12, 2026
<!-- TERMAG_PR_BODY_BEGIN -->
BEGIN_COMMIT_OVERRIDE
feat: Add experimental support for flag overrides, read from local JSON
or YAML files or supplied by a custom override source
feat: Mark evaluations that a flag override affected with
`overrideAffected` in the evaluation reason, and count them under
separate summary counters
fix: Stop the components that already started when the client fails to
start
END_COMMIT_OVERRIDE

Every change in this PR was already reviewed and merged through its own
PR against `feat/overrides`: #525, #526, #527, #528, #529, #530, #572,
#574, #576, #577. This PR merges the branch into `main`.

Flag overrides let an operator force flags and segments to known
definitions on a running client, whether or not the client can reach
LaunchDarkly, and remove the override when the incident is over. They
are an option of the FDv2 data system, on both `LDClient` and
`AsyncLDClient`, with a file-based source in
`ldclient.integrations.overrides.FileOverrideSourceBuilder`, and are
experimental.

The one change outside the feature: #527 stops the components that had
already started when the data system fails to start, so a failed client
construction no longer leaves threads behind.

`main` was merged into `feat/overrides` in 46e5f0e, so the branch has
every `main` commit. That merge resolved two conflicts and made one
follow-up change:
- The capability lists of both contract test services keep both sides:
`flag-overrides` from this branch, and the retry conformance, service
endpoints, wrapper, proxy and TLS capabilities from `main`.
- One async override test used `AsyncHook`, which #548 removed from
`main`. It now uses `Hook`, which the async client accepts.

Before the push:
- the unit test suite passed (2,357 tests, without the Redis, DynamoDB
and Consul store tests, which need servers);
- `mypy`, `isort` and `pycodestyle` were clean;
- the contract test harness's `flag overrides` suite passed against both
the sync and the async test service (88 tests each).
<!-- TERMAG_PR_BODY_END -->

<!-- termag: box=i-0575d1c54853058d7 project=python-server-sdk -->
<sub>Opened from termag</sub>

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Adds **experimental flag overrides** so operators can force known
flag/segment definitions at runtime (via local JSON/YAML files through
`FileOverrideSourceBuilder`, or a custom `OverrideSource`), integrated
into the FDv2 data system for sync and async clients through
`datasystem_config.override_source` / `ConfigBuilder.overrides()`.
> 
> The data system now maintains an **override layer** merged over the
normal store, serves override-only flags when LaunchDarkly data is not
ready, and fans out **dependency-aware flag-change notifications** when
LD data affects overridden prerequisites/segments. Evaluations propagate
an **`overrideAffected`** reason marker; analytics treat those
evaluations differently (summary counters only, `all_flags_state`
disables per-flag tracking for them).
> 
> Contract test services advertise **`flag-overrides`** and build
override sources from harness config; docs add
`ldclient.integrations.overrides`.
> 
> Also **fixes client startup**: if the data system fails to start,
already-started components are stopped so failed construction does not
leak threads.
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
46e5f0e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants